Spotlight not working in Mountain Lion

We have received numerous report that in Mountain Lion Spotlight is not searching local HDDs.  Since Outlook also uses Spotlight, no mailbox searching was possible either.

The fix is two-fold.  The first thing to do is delete the index metadata:

sudo /rm ./metadata_never_index

Secondly, re-enable spotlight indexing:

sudo mdutil -a -i on

Technically simply turning Spotlight indexing ON using the mdutil command should resolve the problem.  However we have found that you must first delete the index file first.

Update:  We ran into a computer where the above steps did not resolve the indexing problem.  On that machine the .metadata_never_index file was in the root volume (don't know why).  Removing that file and then running "sudo mdutil -i on /" resolved the problem.
 
1.  Double check that the .metadata_never_index file is not located at the root of the volume you are attempting to index
2.  Disable spotlight indexing for the volume:  sudo mdutil -i off /
3.  Remove the index from the volume:  sudo mdutil -E /
4.  Remove the .Spotlight directory from the root if it exists
5.  If there is a .Spotlight-V100 directory in the root, remove it: sudo rm -rf .Spotlight-V100
6.  Enable indexing for the volume:  sudo mdutil -i on /

Recon failed during the submit process. Could not recognize the JSS response

After upgrading to JSS 8.6 a site noticed that none of the Jamf binaries on the clients were being updated.

When we attempted to Recon the Macs again the process would fail with a message "Error enrolling computer.  Could not recognize the JSS response."

 
We would receive the same error no matter what method we used to get the computer into inventory: local Recon, Quick-Add Package, command line enrolment, etc.

From the screen-shot it would seem that the problem had to do with the client or server's certificate.  However, after renewing the JSS' certificate, double-checking that it was valid and that existing clients could communicate with the JSS we were at a loss to explain why Recon was failing.

The answer was both simple and completely crazy.  It seems that if there is a non-standard character in the "Display message to User:" field in Restricted Software it causes Recon to fail with the above mentioned error.

The text of the original message:  Este software no está permitido en la red corporativo

After changing "está"  to "esta" we were able to Recon computers normally.

How bizarre is that?

A big thanks to Derek at Jamf support for figuring out the problem.  We would have spent years hunting for a fix on our own.

10.7 and 10.8 Clients do not generate Kerberos tickets at login: FIX

(For information about forcing 10.6.x clients to get a Kerberos ticket at login see this Apple KB)

We have encountered a problem where Lion and Mountain Lion clients are not generating Kerberos tickets at login.  This causes problems with single-sign-on (SSO) to network shares as well as with Kerberos enabled applications like SAP.

Apparently Apple's latest version of Kerberos will not automatically request a ticket when a Kerberos enabled application launches.  Instead you must either go to Terminal and type "kinit" and enter your password or force a ticket to be generated at login using the following instructions.

Note: in order to correctly authenticate using Kerberos enabled applications like SAP your user name in AD and in the application itself must match.  This means that both your SAMAccount name and UPN in must be identical- including the case.  Many non-Microsoft instances of Kerberos are case-sensitive.

To generate a Kerberos ticket at login (10.7 and 10.8 clients):

•  Open Terminal
• Type "sudo -s" and authenticate as super user
• Navigate to /etc/pam.d
• Type "cp authorization authorization.bak" to make a backup of the authorization file
• Type "pico authorization" to edit the file (you could also use vi or your favourite editor)

Find the line:

auth       optional       pam_krb5.so use_first_pass use_kcminit

Add the key "default_principal" to the end of the line. For example:

auth       optional       pam_krb5.so use_first_pass use_kcminit default_principal

• Save the file
• Restart and log back into the computer
• Check that a Kerberos ticket has been generated by opening Terminal and typing "klist"

Hosting Mountain Lion Software Updates on 10.6 or 10.7 Servers

This information came directly from Apple's KB:

1. Update your server to Mac OS X Server v10.6.8 (or 10.7)
2. Stop the Software Update service if it is running.
3. Quit Server Admin if it is active.
4. Update /etc/swupd/swupd.plist to begin hosting OS X Lion software updates. (See details below.)
5. Update /etc/swupd/swupd.conf to allow OS X Lion computers to receive updates. (See details below.)
6. Open Server Admin and start the Software Update service.
7. Use the instructions in Mac OS X Server v10.6: Using the Software Update service with multiple Mac OS X client versions to point your OS X Lion clients to this server.

In step 4, you will need root access to update the file /etc/swupd/swupd.plist. To be safe, make a backup copy of the file before editing it. Add the string in bold below to the otherCatalogs array:

    otherCatalogs
   
        index-leopard.merged-1.sucatalog
        index-leopard-snowleopard.merged-1.sucatalog
        index-lion-snowleopard-leopard.merged-1.sucatalog
   

For Mountain Lion change the string to:

index-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog

In step 5, you will need root access to update the file /etc/swupd/swupd.conf. To be safe, make a backup copy of the file before editing it. Locate the following line near the end of the file:

    RewriteCond %{HTTP_USER_AGENT} Darwin/11

Edit the following line to read:

    RewriteRule ^/index\.sucatalog$ /index-lion-snowleopard-leopard.merged-1.sucatalog

For Mountain Lion change the string to:

RewriteRule ^/index\.sucatalog$ /index-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog

Troubleshooting Mac Binding and Authentication Problems: AD perspective

Ensuring a clean DNS and directory environment is key to resolving a host of problems, not all of them Mac specific.  Many binding, authentication and slow login issues can be traced back to incorrectly configured DNS, a missing subnet in Sites and Services or poor replication between domain controllers.

This post is designed to provide a check list of sorts for directory services administrators as they attempt to diagnose Mac and AD issues.  None of the steps listed here are designed for end-users or the Mac clients themselves- they are back-end checks for system administrators comfortable with DNS, DHCP and domain controller diagnostics using the command line.

Domain Controller Checks

The health of the local DC is a factor with Mac binding and post-binding authentication problems.  Many sites have ageing DCs with inadequate RAM and they may not be able to process authentication request properly.

Adding to the problem is the fact that pre-Lion Macs are only dimly site-aware.  This means that during the binding process they can attempt to connect to whatever DC they find that has a valid SRV record.  Thus, even if a local DC is brand new and functioning perfectly, Macs could still have binding problems because they are looking to a DC on a remote site that is not reachable or otherwise unavailable.

Steps can be taken on the client to force the use of the local DC: using a custom edu.mit.kerberos file and setting the local DC as the preferred domain controller during the binding process.  These procedures are detailed in other posts on this site.

From a Directory Services standpoint there are several checks that can be done to ensure that the DC is performing optimally.

• Check the RAM.  1GB is generally not enough
• Check the processor load to ensure the CPU is not consistently running over 50%
• Check that the primary DNS is set to the DC’s IP address
• Check that the secondary DNS is correct
• Remove WINS from the NIC configuration
• Run a dcdiag to check the general health of the DC
• Ensure that the DC is not multi homed.  If NIC teaming is enabled, ensure that it is configured correctly.
• Run “repadmin” to ensure that the DC is replicating properly and that there are no latency issues

DNS Checks

Mac binding is heavily dependent on a pristine DNS environment.  Duplicate A or PRT records cause major problems.  Additionally, Mac clients prior to 10.7 Lion do not reliably register PTR records.  Without these reverse records Macs can fail to bind and may experience authentication issues.

The first step in any troubleshooting process is to check for duplicate A and PTR records.  Our organization enables PTR record scavenging; this has caused the instance of duplicate PTR records to drop dramatically.  However, duplicates can and do creep in and enabling A record scavenging is not recommended .  A full clean-up involves  removing all workstation A and PTR records and starting fresh.

• Connect to DNS on the local DC (or open the DNS mmc snap-in)
• Expand the Reverse Lookup Zones
• Find the zone or zones for the site
• Highlight and delete all existing PTR records
• Do not delete PTR records for non-Windows servers or network devices
• Expand the Forward Lookup Zones
• Delete the A records which belong to desktops and notebooks only!

Because of changes in the DNS snap-in, it is no longer possible to easily search for PTR records by subnet.  Therefore in order to find and delete errant PTR entries you must use the DNS services on a 2003 DC or the XP DNS mmc snap-in.

The DHCP server must be set to update A and PTR records as well as grant ownership of the records to a service account.  These changes are detailed in the “DHCP Checks” section of this page.

SPN
Check Service Principle Names (SPN) using setspn -l [server short name]

In particular you are looking to see that the ldap, DNS and HOST entries are present and have both the short and FQDN of the local DC.

SRV
Check for missing or inaccurate SRV records.

• Open DNS snap-in
• Go to Forward Lookup Zones
• Expand the domain
• Expand “_sites"
• Locate the site you are working on
• Expand the location and click on “tcp”
• Ensure that the correct _kerberos and _ldap entries are present

Most locations should have only one _kerberos and _ldap entry and they should be directed to the local DC.  Some sites have many entries; it has been found that multiple entries can cause slow logins for both PCs and Macs and also prevent Macs from binding/authenticating correctly.  Unless there is a specific requirement, SRV records in this location should be kept to an absolute minimum.

Conversely, if the site is missing _kerberos or _ldap entries they need to be added.  Check the following locations to ensure that the local DC’s SRV records are present:

• Open DNS snap-in
• Go to Forward Lookup Zones
• Expand the domain
• Click on “_tcp”
• Check that the local DC has both _kerberos and _ldap entries. If not, add them
• From the domain level expand “_msdcs”
• Expand “dc”
• Open “_tcp” and check that the local DC has both _kerberos and _ldap entries. If not, add them

SRV records for the DC should be created each time Netlogon service starts however, sometimes this must be forced.

• On the DC open a terminal window
• Run the command “nltest /dsregdns” to force the DC to register it’s SRV records in DNS

nltest is also useful for checking and repairing secure channel communications issues.  A full list of nltest commands can be found HERE.


DFS
Check that DFS is active on the local DC.

• Go to Start/Run
• Type  \\your.domain.here\sysvol
• Right-click anywhere in the folder window
• Got to Properties/DFS
• The active DFS patch should be the local DC

If the local DC is not the active DFS path follow these instructions to correct the problem.

Sites and Services

Like Windows computes, Macs use Sites and Services to ensure that they are authenticating to the correct DC.  Get a list of subnets from the location and ensure that they are entered into Sites and Services correctly.  Make sure the list includes any Wi-Fi subnets that are on the IPG network as well as server subnets.

Double check that the correct DC is setup and that its replication partners are correct.  It is also a good idea to force replication (you may have already done this using “repadmin” in an earlier step).

DHCP Checks

DHCP must be configured in such a way that Mac clients are forced to register A and PTR records when they receive a DHCP lease.

The site must be running a Windows 2003 or later DHCP server in order to make these configuration changes.  The process will not work with Windows 2000 DHCP.

• Open DHCP manager either through the mmc snap-in or on the DHCP server
• Right click on the server and select Properties
• Click on the DNS tab
• Put a tick in the following boxes
• Enable DNS dynamic updates according to the settings below:
• Always dynamically update DNS A and PRT records
• Discard A and PTR records when lease is deleted
• Dynamically update DNS A and PTR records for DHCP clients that do not request updates
• Click on the Advanced tab
• Click the Credentials button

Here you should enter the name of a service account that will own the DNS records.  You can make the account name anything you like but its purpose should be apparent, "service.dns" or "service.dnsrecords" etc.  Create the account on AD and set the password to never expire.  This account will be used by DNS to add and delete records when machines get new leases or old leases expire.  Without this account records can remain in DNS; if a Mac client attempts to use an IP that is already associated with a DNS record the client can fail to bind.

• User name: your DNS service account
• Domain: netbios name for the domain
• Password: enter the service account password
• Click OK and OK again to save the changes 

Further DHCP checks

• Remove WINS from the DHCP scope
• Check that the DHCP scopes match what is in Sites and Services
• Check that the FQDN of the domain is correct
• Check that the primary and secondary DNS servers are correct
• Look at the DHCP leases and delete any that do not have a resource name associated with them
• Check that there are enough free IP addresses

User Account Checks
Once a Mac client has been bound to AD it can experience a host of authentication issues.  These normally manifest themselves as a “shaky login”: when a user attempts to log in the authentication window shakes and denies them access. 

Although most of the troubleshooting steps regarding shaky logins involve client-side procedures, there are a couple of things you can check in AD.

• Remove the UNC path to the user’s home folder in their AD profile
• Ensure that the SAMAccount name and UPN match and are in all lower-case

Conclusion
DNS issues cause the vast majority of Mac binding and authentication issues.  If a site is having across the board problems in these areas, performing DNS health checks should be Directory Services’ first troubleshooting steps.

Restoring Windows shares after migrating data from one server to another

Situation:  An office used robocopy to copy files from a Windows 2003 file server to a Windows 2008 R2 server.  Permissions were copied but shares were not.

Follow the steps below to restore the file shares.  Note: the drive letters and paths must be the same on the two systems or this procedure will not work.

This procedure applies only to NetBIOS shares and not to Macintosh volumes.

1.  On the SOURCE server that contains the shares that you want to transfer launch regedit
2.  Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanServer\Shares
3.  Export the registry key by going to the File menu and clicking on Export
4. Save the file to a location that is accessible from the new server
5. On the TARGET server launch regedit
6. Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanServer\Shares
7. Import the saved registry key by going to the File menu and clicking on Import
8. Enter the path to the saved registry key that you exported in step 3
9. Restart the server

 Full details of this procedure can be found in this MS KB article.

OS 10.7 Lion and Citrix: 100% CPU use after upgrade

After upgrading from 10.6.8 to 10.7.4 I noticed that my battery life plummeted.  Prior to the upgrade I would be getting a solid 5 hours; after installing Lion my battery life dropped to 1:15.  Applications were also performing very poorly and the entire system seemed sluggish.

After some investigation and speaking with Apple I discovered that the problem was with the Citrix Access Gateway plug-in.  I removed it and my system instantly returned to normal.  In fact I started getting six hours of battery life so it could have been causing problems when I was on 10.6.8 too.

Citrix has acknowledged that several of their Receiver components are having problems with Lion.  HERE is a link to the Citrix page that explains the problems and offers work-arounds.

SAP launch error in Lion

On AD bound Lion (10.7.3) Macs were were receiving the following error on launch:

Unable to load GSS-API dyld Shared Library
named "sncgss.dyld"

We have seen similar errors with Snow Leopard (10.6.x) machines and it normally indicated that 32bit mode was not enabled; this was not the case in London.

The fix we found involves setting the SNC_LIB varriable using the following commands in Terminal:

echo Setting the SNC environment for current user.
mkdir ~/.MacOSX >/dev/null 2>&1
defaults write ~/.MacOSX/environment.plist SNC_LIB "/usr/lib/libgssapi_krb5.dylib"

Strangely, this did not fix the problem right away- we had to wait for upwards of an hour for the fix to "take"; once it did the users could launch the SAP client normally.

Office 2011 SP2 Database Problems: Workaround from Microsoft

Microsoft is acknowledging that many customers are having problems with the Outlook database after applying SP2.  They have posted the following advice and workarounds:

http://blog.officeformac.com/office-for-mac-2011-sp2-database-upgrade-workaround/

Office 2011 14.2 update installation problems

We have received numerous reports of problems when attempting to install the Office 2011 SP2 (14.2) update remotely using Casper.  However, the update seems to deploy using ARD but you should be aware of a few peculiarities. 

Microsoft changed the format of their update package from an .mpkg to a .pkg.  While this might not sound like much of a change there are significant differences in how the two types work. .pkg files are flat packages that don't do anything more than bundle up files into an installation wrapper.  .mpkg files are much more customizable and can link to other packages.

It seems that even though the 14.2 updater is a .pkg it is not actually a flat file.  I don't know what magic Microsoft has tried to work but there seems to be many problems relating to this new composition.

While the updater runs fine from a local machine, attempting to deploy it though Casper does not work.  Even though Casper (and the Mac) report that the installation was successful, the Office version remains the same and the update does not get installed.

Installing the .pkg through ARD works, however when you launch Outlook you get the following error:

Deleting the daemon from the user's startup items and restarting fixes this problem.

If you launch Word before restarting you will probably see this error:

After a restart this error should go away.

Once you restart, launch Outlook and you will get the following message:

Before clicking "Upgrade" it is a good idea to back-up the user's Main Identity.

Once you click "Upgrade" you are committed- you must let it finish or you will get data corruption.  A 2GB profile on an i5 iMac took about 7 minutes to upgrade.

The upshot of all this is that SP2 is not a "fire and forget" update.  A good deal of intervention is required to ensure that the process goes smoothly.

Deploying McAfee ePO agent using Casper

Jamf has an excellent KB article on how to deploy the McAfee ePO agent.

https://jamfnation.jamfsoftware.com/article.html?id=182

However, if you have already installed the unmanaged client you will need to remove the agent before reinstalling.

1)  Uninstall the existing (stand-alone) agent using Casper Remote by running the following command:

/Library/McAfee/cma/uninstall.sh

2)  Install the agent using Casper Remote and the following command:

 /Library/Application\ Support/McAfee/install.sh -i

If you do not first uninstall the agent you will receive the following error:

"An higher or same version of the agent is already installed installer: Error - The installed version of the agent is already greater than or equal to this version. Installation cannot continue"

ExtremeZIP Optimisation and General Settings

We have received several requests as to the best way to optimise performance of Windows servers running ExtremeZIP file sharing for Apple clients.  In particular many sites see the following warnings in their ExtremeZIP logs (Windows 2003 server):

If you see these warnings, or even if you don't but still want to make sure your system is running as smooth as possible, the first thing you should do is read this KB article from Group Logic:

http://support.grouplogic.com/?p=1544

It goes into detail about the registry settings that need to be tweaked in order to get the best out of ExtremeZIP.  The KB is from 2009 but the registry changes are still applicable on 2008 R2 servers.  Here is an example of the registry changes to make:

It is also recommended that on Windows 2003 server you Maximize Data Throughput for Network Applications instead of "File Sharing".  To do this follow these steps:

1. Right-click on My Network Places and select "Properties"
2. Right-click on your NIC and select "Properties"
3. Highlight "File and Print Sharing for Microsoft Networks" and click the "Properties" button
4. Enabled Maximize data throughput for network applications 

General ExtremeZIP Settings

We have found that the following setup works well for most locations (your mileage may vary).

Launch ExtremZIP Administrator and click on the "Settings" button; select the "File Server" tab.

Put ticks in the "Allow Encrypted Logins" and "Allow Kerberos Logins" only.

Click on the "Security" tab.

In order to have ExtremeZIP support Unix and more granular ACLs on volumes you must supply an AD account.

1. Put a tick in "Global Catalog"
2. Enter an account that has domain access.  It is a good idea to use a service account that has a password set to never expire.  Add the account in the format of "yourdomain\account.name"
3. Enter the account password
4. For "Domain" enter the FQDN of your domain
5. For "Permissions" use the the settings in the above example.  You generally don't want your Mac clients to be able to change folder permissions so remove the tick from "Allow Mac clients to change permissions"
6. If you like you can select one of the tick boxes under "Show only accessible:"  If you check the "Folders" option, users will see only folders that they can access. If you check the "Files" option, users will only see files they can access
7. Under "Other Options" you can "Allow remote administration of server" if you want Windows admins to be able to use the remote admin features of ExtremeZIP (click the "Help" button for more details)
8. By putting a tick in "Notify Mac clients of password expiration in XX days" your Mac clients will see a password change notification when they connect to an ExtremeZIP enabled share.  Note: they can not change their password directly from the expiration warning dialogue box.

Click on the "Search" tab.

Search settings should be customised for your environment.  If you put a tick in "Index volumes for search" ExtremeZIP will create an index of all the files on your ExtremeZIP volumes.  This allows Mac clients to do rapid searches of files/folders using the ExtremeZIP index and does not impact the server drives.  However, this only applies to searches performed at the root level of a volume.  Searches performed in sub-folders will use standard Windows enumerated searching (more disk intense).

Keep the default "Maximum search index cache size" at 20MB.

"Lazy Indexing" limits the amount of system resources available for indexing and is recommended for servers under high load or with considerable I/O traffic.

"Automatically rebuild sparse indexes" is a maintenance function of ExtremeZIP that cleans up the indexes when 1/3 of the records are stale. 

If you want to enable Spotlight searching put a tick in the "Support Spotlight Search" box.

Click on the "Filename Policy" tab.

If you are working in a mixed environment where Mac and PC users must share files it is recommended that you use ExtremeZIP's filename policy to force Mac clients to create files/folders that are compatible with Windows systems.

To enable filename policy enforcement put a tick in "Enforce Filename Policy".

"Apply policy to all volumes" will enable the policy on all files/folders in newly created volumes.  Legacy volumes and existing files/folders will not be affected.

If you want the user to simply be warned of naming violations un-tick "Reject policy violations..."

Under "Do not allow:" you can select what characters and name lengths are acceptable.

Put a tick in "Characters illegal in Windows file or folder names" if you want to exclude filenames that contain characters not allowed in Windows.  The characters are / ? < > \ : * |  trailing spaces and trailing full-stops.

Put a tick in "Characters that will not display in Windows Explorer" to exclude names that can not be displayed in the font used by Windows Explorer (the default is Tahoma).

By limiting the file/folder names to 254 characters you can ensure compatibility with Windows systems.  You may also enable the 254 character limit for paths as well.

Click on the "Service Discovery" tab.

For "Server Name" enter the server name as you want it to appear to Mac clients.

If you are in a standard Windows/Mac environment you only really need to tick the "File" and "Print" boxes next to "Bonjour" to make the ExtremeZIP enabled Windows server visible in a search from Mac clients.  The rest of the protocols only need to be enacted if you use them at your location.

Note: there should be no need to use AppleTalk unless you have OS9 clients or AppleTalk printers.

The latest information, knowledge base articles and manuals can be found on the GroupLogic site.

Slow Login on Snow Leopard Clients Bound to AD

We have received several reports that Snow Leopard clients that are bound to AD are experiencing very slow logins- up to five minutes before the desktop would appear.

One fix is to remove Active Directory/All Domains from the Authentication tab in Directory Services.  Once this is done the login is very fast.

The drawback is that no other user will be able to log into the computer using their AD credentials.

AD password changes done via the login screen or System Preferences/Users/Change Password continue to work.

Full details can be found on Tech Smog.

How to enable or disable Airport from the command line

I was asked how to disable the Airport from the command line.  Here is what you do:  

Disable:  sudo ifconfig en1 down 

Enable:  sudo ifconfig en1 up 

"en1" is the network interface.  "en0" is usually Ethernet and "en1" is normally the Airport.

You can send these commands via ARD.

Casper Admin Can Not Mount a Distribution Point

Problem:  When attempting to Launch Casper Admin the distribution point will fail to mount and an error is displayed saying "The master Distribution Point (server name) could not be mounted".  It then gives you some tips to resolve the issue.

Solutions:  Follow the tips and check that the correct permissions and passwords are set on the Distribution Point's file share.

One good way to check if the permissions are set correctly is to connect to the server share from another Mac using the "casperadmin" and "casperinstall" accounts.  If you can mount the Distribution Point then the permissions should be OK.

With the Distribution Point mounted launch Casper Admin.  If it opens without an error then your problems are very likely not related to permissions but rather a corrupt or missing symbolic link to the CasperShare folder.

In /Library/WebServer/Documents there is a folder called CasperShare that is a symbolic link to the "real" CasperShare.   If you suspect that the link is corrupt the easiest thing to do is simply delete the symbolic link and create a new one. 

Note: in Casper versions prior to 8.31 you could use the JSS Setup Utility to recreate the symbolic link.  Unfortunately since version 8.31 the JSS Setup Utility has been removed and you have to do the process manually.

To recreate a symbolic link to the CasperShare do the following:

• Open Terminal
• Type "cd/"
•  Type "sudo ln -s /Shared\ Items/CasperShare/ CasperShare" this will create a symbolic link to the CasperShare and put it in the root of the drive
•  Move the CasperShare symbolic link to /Library/WebServer/Documents and replace any existing CasperShare link
• If you do an "Apple-I" to get information on the Symbolic Link you will see that next to "Original" it has the path to the CasperShare

 

After you have recreated the symbolic link launch Casper Admin to test connection to the share.

Remember, you can not launch Casper Admin from the server that hosts the primary Distribution Point.

For a good overview of symbolic links see this article.

Where are JSS database backups located?

sudo /private/var/backups/jss/

How to build a Casper netboot set using Apple's System Image Utility

Information directly from the Jamf site:

https://jamfnation.jamfsoftware.com/article.html?id=64

The instructions tell you how to create a netboot set that will automatically launch the Casper Imaging Utility.  However if you want to use your netboot set in different environments then you should not auto-launch Casper Imaging.  Rather, just put the icon in the Dock and enter the JSS info when prompted.  It's one more step but worth it if you need to share your netboot set with other sites.

High CPU usage caused by: krb5kdc: didn't find any realms when starting

We have seen several instances where a computer is unable to launch applications and suffers other performance problems when edu.mit.kerberos.krb5kdc repeatedly exits and respawns.  Look in the Console log for these errors:

To fix this, open Terminal and type:

sudo /usr/libexec/configurLocalKDC

It has also been suggested that you delete all files in the following location:

sudo /var/db/krb5kdc/

Although some people may not have any files in that location.

dyld: shared cached file was build against a different libSystem.dylib, ignoring cache

On a Mac 10.6.8 server we were experiencing poor performance and erratic behavior from Server Administrator.  Looking at the Console Logs we found that it was filled with the following error:

dyld: shared cached file was build against a different libSystem.dylib, ignoring cache

Apparently this is a fairly common error in 10.6.x and can affect a wide variety of applications.  The fix is pretty straightforward.  Open Terminal and type the following:

sudo update_dyld_shared_cache -force

Windows 2008 R2: Members of the local "Administrator" group do not have admin rights to shares

We have received many reports of a problem with Windows 2008 R2 servers where shares containing the local "Administrator" group were not accessible by members of that group.

For example: we have a GPO that makes our "Domain Admin" group a member of the local Administrator group of all our servers.  However, when a Domain Admin would log onto a server he would not have access to server shares.

We resolved the problem by  disabling User Account Control (UAC) on the server.  This Microsoft KB article describes how to turn UAC off in Server 2008 R2.

For a full overview of UAC and how to turn it off/on on other servers see this KB.

Note: these changes require a restart.

Single-Sign-On (SSO) not working for Snow Leopard clients connecting to a Windows server running ExtremeZIP

We received a report from an office that three of their Windows 2003 servers running ExtremeZIP were not allowing SSO connections from AD bound Snow Leopard Macs.

After a good deep-dive into the problem, including packet traces and help from Group Logic, we resolved the problem.  Here are the steps we took:

Make sure the Mac clients are using the FQDN to connect to the ExtremeZIP AFP volume on the server.  Short names should not be used (in Lion you must use the FQDN or you get an error).

Check that the time on the server, clients and DC match.  One of the servers' clock was out by six minutes (max Kerberos time skew is five minutes).  When the time was set correctly Lion clients were able to log in.

Check that the Server Principle Name (SPN) of the servers is correct; if they are not then authentication can fail.  Read more about SPNs here.

To check the SPN on a Windows 2003 server you must first download and install Windows Server Support Tools.  You can get them here.

After you have installed the tools go to Programs/Windows Server Support Tools and launch the app- it will open a command line.

Both the long and the short SPN for the AFP protocol need to exists for your servers:
afpserver/servername.company.com
afpserver/servername

To display the SPNs from the Support Tools command line type "setspn servername"

You should see both the FQDN and the short name.  If one is missing do the following:

- To add the long name: setspn -a afpserver/servername.company.com servername
- To add the short name: setspn -a afpserver/servername servername

We also found that although the Snow Leopard clients were authenticating users correctly, they were not generating a Kerberos ticket at login (you can verify this by going to the Ticket Viewer.app located in System/Library/Core Services).  After manually generating a Kerberos ticket, SSO worked.

To force a Snow Leopard client to generate a Kerberos ticket at login follow the instructions in this Apple KB article.

After carrying out each of these steps, the Snow Leopard clients were able to get SSO to the ExtremeZIP enabled servers.

Although it wasn't necessary in this case, make sure you update ExtremeZIP to the latest version

How to check if an Apple server is Kerberized against AD: verify Service Principals

If Mac clients are having trouble accessing a bound OS X server, check that the server is Kerberized against AD.  First, run the following command:

sudo klist -kt

You should see a number of service principals with the Kerberos realm of your.domain.com

Second, you need to ensure that the correct service principal is in use by the AFP service.  You can use the following command to do this:

sudo serveradmin settings afp:kerberosPrincipal

This should show something like "afpserver/@YOUR.DOMAIN.COM".   If it shows a value in the LKDC realm it is incorrect and will need to be fixed before you can connect using Kerberos.

Here's a command you can use to fix it:

sudo serveradmin settings afp:kerberosPrincipal = "afpserver/@YOUR.DOMAIN.COM"

Using Snow Leopard server for Lion software updates

Here is a step-by-step guide for setting up a Snow Leopard SUS server so that it will distribute Lion updates.

I do not know who originally wrote this article.  If anyone can find the author please let me know and I will credit him/her.


First things first. Fire up Server Admin and stop the Software Update service. Next fire up a Terminal window and head to /etc/swupd.

cd /etc/swupd

Now let's make backups of the .plist and .conf files for swupd.

cp swupd.plist swupd.plist.bak; cp swupd.conf swupd.conf.bak

Great. Now we are going to add a catalog to the catalog array in the swupd.plist. When we are done it will look like this:

otherCatalogs
   
    
       index-leopard.merged-1.sucatalog
       
       index-leopard-snowleopard.merged-1.sucatalog
       
       index-lion-snowleopard-leopard.merged-1.sucatalog
   
    

You can do this with a text editor, but PlistBuddy makes it easier.


sudo /usr/libexec/PlistBuddy -c 'add :otherCatalogs:2 string index-lion-snowleopard-leopard.merged-1.sucatalog' /etc/swupd/swupd.plist

Will do it in one shot. Adding this catalog is what will tell our SUS Server to go and get the Lion updates.

The next bit we need to do is to tell the server that it can provide this catalog to Lion clients. Open up swupd.conf with the editor of your choice.

vi swupd.conf

Go to the bottom of the file, or just search for Rewrite. Look for the Darwin/11 agent string and change the rewrite rule to look like this.

RewriteCond %{HTTP_USER_AGENT} Darwin/11
RewriteRule ^/index.sucatalog$ /index-lion-snowleopard-leopard.merged-1.sucatalog

Basically we are adding the word "lion" into the sucatalog name.

Now close out of the terminal and start up the Software Update Service with Server Admin again. If you watch the logs, or the updates tab, you will see the Lion updates and on-demand software appear and begin downloading. After a while they will be local and you can start updating those Lion clients!

How to enable Directory Services debugging and packet capture at log on

When diagnosing login problems it can be very helpful to generate a packet capture during log-in.  Unfortunately tools like Wireshark or PacketPeeper do not run at start-up. 

Following are the commands to enable DS debugging and a packet capture during log-in.  You will need another computer connected via SSH to the one on which you want the packets captured.

1. Run the following command to set the debug level to seven (all one line):
       
 sudo defaults write /Library/Preferences/DirectoryService/DirectoryServiceDebug "Debug Logging Priority Level" -integer 7

NOTE: Only run this command if you are running 10.5 or later

2. Enable Directory Service Debugging by running the following command:
        
        sudo /usr/bin/killall -USR1 DirectoryService
       
3. SSH to the client and start a packet capture:
       
        sudo /usr/sbin/tcpdump -vvv -n -s 0 -w /Library/Logs/`date +%Y%m%d-%H%M%S`.pcap
               
4. Reproduce the issue by attempting to login.

5. Stop the packet capture using Control+C.

6. Disable Directory Service Debugging by running the following command again:
        
        sudo /usr/bin/killall -USR1 DirectoryService

The capture will be in the /Library/Logs with a ".pcap" extension.