Open-AudIT

What's on your network?
It is currently Wed Jan 17, 2018 3:22 pm

All times are UTC + 10 hours




Post new topic Reply to topic  [ 30 posts ]  Go to page Previous  1, 2
Author Message
PostPosted: Sat Mar 17, 2012 6:45 pm 
Offline
Site Admin
User avatar

Joined: Mon Jun 07, 2004 11:48 am
Posts: 1933
Location: Brisbane, Australia
Have removed CheckForHungWMI and LogKilledAudit from audit_domain and audit_list scripts.

_________________
Support and Development hours available from Opmantek.
Please consider a purchase to help make Open-AudIT better for everyone.


Top
 Profile  
Reply with quote  
PostPosted: Tue Mar 20, 2012 1:51 am 
Offline
Moderator

Joined: Fri Jul 20, 2007 8:27 am
Posts: 1223
Mark wrote:
I wonder if i can test if a column exists first? Will have to look into that - it would make upgrading the DB "safer"...

Bad news on this one. I reviewed my previous problem with the update database routine and I must have had a change in my database that caused an error with the upgrade routine. Looking at that code I see that the upgrade tries to wrap all the changes in a transaction for just this occasion. It looks like this won't work because ALTER TABLE is of a class of statements that cause an implicit commit so pretty much the only thing rolled back is the oa_config updates.


Top
 Profile  
Reply with quote  
PostPosted: Tue Mar 20, 2012 2:13 am 
Offline
Moderator

Joined: Fri Jul 20, 2007 8:27 am
Posts: 1223
The trunk OAv2_mysql.sql needs updated final oa_config INSERTS. It still has beta4 in there.


Top
 Profile  
Reply with quote  
PostPosted: Tue Mar 20, 2012 3:07 am 
Offline
Moderator

Joined: Fri Jul 20, 2007 8:27 am
Posts: 1223
I dislike the empty local domain "Table does not exist error" so simple fix.

Code:
if (domain_array(0) = "") and (local_domain > "") then
   domain_array(0) = local_domain
elseif (local_domain > "") then
   number_of_domains = ubound(domain_array)+1
   redim Preserve domain_array(number_of_domains)
   domain_array(number_of_domains) = local_domain
end if


This doesn't handle an empty domain_array and blank local_domain.


Top
 Profile  
Reply with quote  
PostPosted: Tue Mar 20, 2012 7:09 am 
Offline
Site Admin
User avatar

Joined: Mon Jun 07, 2004 11:48 am
Posts: 1933
Location: Brisbane, Australia
DB transactiions - well damn :-(
Updated the mysql definition file.
Updated the audit_domain file.

_________________
Support and Development hours available from Opmantek.
Please consider a purchase to help make Open-AudIT better for everyone.


Top
 Profile  
Reply with quote  
PostPosted: Tue Mar 20, 2012 7:21 am 
Offline
Moderator

Joined: Fri Jul 20, 2007 8:27 am
Posts: 1223
I'm seeing pretty high cpu utilization while auditing software and services and have been seeing this since OAv1. I suspect (but don't know) it is the escape_xml function and vbscript's horrendous string concatenation performance. When I've got a domain audit running remote audits the audit server's CPU is basically pegged at 100%.

Since the code is littered with stuff like this:
Code:
result = result & "<system>" & vbcrlf
result = result & "   <sys>" & vbcrlf
result = result & "      <system_timestamp>" & escape_xml(system_timestamp) & "</system_timestamp>" & vbcrlf
result = result & "      <system_uuid>" & escape_xml(system_uuid) & "</system_uuid>" & vbcrlf
result = result & "      <system_hostname>" & escape_xml(system_hostname) & "</system_hostname>" & vbcrlf

Wouldn't it make more sense to take the individual escape_xml calls out of the code and just run a more efficient escape_xml over the final "result" before sending it up to the server. You could then rewrite the escape_xml using something more efficient like ADODB.Stream like in urlEncode.

Just a thought and there could be unintended consequences as I haven't really thought hard about this.


Top
 Profile  
Reply with quote  
PostPosted: Tue Mar 20, 2012 7:29 am 
Offline
Site Admin
User avatar

Joined: Mon Jun 07, 2004 11:48 am
Posts: 1933
Location: Brisbane, Australia
If we don't escape the entire result variable until the end, we'll end up escaping the XML attributes, as well as the variable content... (all the < and >).

And yeah - my audit server's run at 100%.

I suspect it's to do with parsing the installed software and parsing the logs in order to find out WHO and WHEN the software was installed. Am open to suggestions...

_________________
Support and Development hours available from Opmantek.
Please consider a purchase to help make Open-AudIT better for everyone.


Top
 Profile  
Reply with quote  
PostPosted: Tue Mar 20, 2012 7:30 am 
Offline
Moderator

Joined: Fri Jul 20, 2007 8:27 am
Posts: 1223
See - not much thought. :oops: I wonder if all the string concat can be sped up in another way? Maybe something like this to get rid of all the "result = result & ..." stuff.


Top
 Profile  
Reply with quote  
PostPosted: Tue Mar 20, 2012 9:03 am 
Offline
Site Admin
User avatar

Joined: Mon Jun 07, 2004 11:48 am
Posts: 1933
Location: Brisbane, Australia
My thoughts are that if we run the audit on the remote system, all this is alleviated.

I need to knock out a small shim that sits between audit_domain & audit_list and audit_windows.

When you call audit_domain or audit_list, and you specify audit=remote, instead of directly calling audit_windows, they will call audit_remote (up to the number of concurrent audits). If we put the functionality directly into audit_domain or audit_list, it will run sequentially and not XXX number of times concurrently.

So, audit_remote would first copy audit_windows to the $admin share on the remote system, then make use of PSEXEC or the other .exe (can't remember it's name) to run the audit on the remote machine (and possibly include the self_delete flag). Then it disconnects and (from an audit server point of view) is finished. That way, there is processing only occurring on the remote system, not on the audit server.

Thoughts?

_________________
Support and Development hours available from Opmantek.
Please consider a purchase to help make Open-AudIT better for everyone.


Top
 Profile  
Reply with quote  
PostPosted: Tue Mar 20, 2012 10:07 am 
Offline
Moderator

Joined: Fri Jul 20, 2007 8:27 am
Posts: 1223
Mark wrote:
I suspect it's to do with parsing the installed software and parsing the logs in order to find out WHO and WHEN the software was installed. Am open to suggestions...

I watched cscript during an audit using Process Explorer and the CPU usage spike at the end of the script doing Software and Services audits. The Working Set for my example got up to 76MB. I think for some machines we're really running into VBScript problems with all the "result = result &" concatenations. I couldn't see any evidence of the installed_by stuff causing problems.

So I did a super hacky search and replace of the "result = result &" stuff with some ADODB.stream replacement and ran it again. My test audit went from 35 seconds, 78 MB max working set and 100% max cpu to 20 seconds, 15 MB max working set and 20% max cpu.

Super hacky with my debugging stuff still in there. Take the idea and not the code. :lol: Although the output files were nearly identical between the two runs. Did not test uploading.


Attachments:
audit_windows_test.vbs.txt [242.1 KiB]
Downloaded 108 times
Top
 Profile  
Reply with quote  
PostPosted: Tue Mar 20, 2012 10:11 am 
Offline
Site Admin
User avatar

Joined: Mon Jun 07, 2004 11:48 am
Posts: 1933
Location: Brisbane, Australia
Will check it out today (I have the day off from work).
Thanks for the input JPA, much, much appreciated.
What do you think of my "remote" audit plan?

_________________
Support and Development hours available from Opmantek.
Please consider a purchase to help make Open-AudIT better for everyone.


Top
 Profile  
Reply with quote  
PostPosted: Tue Mar 20, 2012 11:21 am 
Offline
Moderator

Joined: Fri Jul 20, 2007 8:27 am
Posts: 1223
Mark wrote:
What do you think of my "remote" audit plan?
Sounds good to me. Although our operation is pretty small so we get away with a central audit server performing the audits. I really should be using remote to audit our remote machines as we've got pretty terrible network bandwidth in our central office at the moment. We do use the randomization feature of OAv1 so we don't audit all the machines in a particular office at the same time (hopefully.) I guess the randomization and parallel auditing is why we don't use remote the way it is now. So your changes would be useful.

Even though the remote audit spreads the load around the current audit script still spikes the CPU. The more software and services you have the more ram and CPU the audit script uses. Basically it's about how much stuff you're returning in the audit. Maybe my earlier idea can make this even less noticeable in both remote and local mode.

PAExec is a redistributable replacement for PSExec.


Top
 Profile  
Reply with quote  
PostPosted: Sat Mar 24, 2012 1:36 pm 
Offline
Moderator

Joined: Fri Jul 20, 2007 8:27 am
Posts: 1223
Per this thread there are a few instances where getkey is called with a second parameter of 2 where it should be 1. The 2 is only used for Office 2010. The trunk currently has it incorrect for Office XP 64bit, Office 2003 64bit and Office 2007 64bit. FYI in case you missed the thread.


Top
 Profile  
Reply with quote  
PostPosted: Mon Mar 26, 2012 10:02 pm 
Offline
Site Admin
User avatar

Joined: Mon Jun 07, 2004 11:48 am
Posts: 1933
Location: Brisbane, Australia
Have pushed the result.WriteText and Office 2010 changes into Bazaar / Launchpad.

_________________
Support and Development hours available from Opmantek.
Please consider a purchase to help make Open-AudIT better for everyone.


Top
 Profile  
Reply with quote  
PostPosted: Tue Mar 27, 2012 4:38 am 
Offline
Moderator

Joined: Fri Jul 20, 2007 8:27 am
Posts: 1223
Thanks. I did test a little more thoroughly and we still get a CPU usage spike at the end because of the URLEncode or the string concatenations but it's quick. I wasn't seeing this CPU spike before because I was just creating output files and not uploading which doesn't do the URLEncode.

I also saw that I forgot to handle the second http request if the first fails.

Code:
objHTTP.Send "form_systemXML=" + result


should be

Code:
result.position = 0
objHTTP.Send "form_systemXML=" + result.ReadText()


There is a vbcrlf tacked on at the end of the result data in the first Send routine. Is this necessary? It isn't in the second send post data so if it's necessary you'd need to add that to the code above.


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 30 posts ]  Go to page Previous  1, 2

All times are UTC + 10 hours


Who is online

Users browsing this forum: No registered users and 6 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group