MSSQL Count

I am trying to write a query that counts records from two tables, but am
getting wrong counts. Basically what i want is this:
SELECT Count(callSched.callID) as callCount, Count(visitSched.visitID) as
visCount, employees.workerNumber, employees.workerName
From employees LEFT OUTER JOIN
callSched on employees.workerNumber = callSched.empID LEFT OUTER JOIN
visitSched on employees.workerNumber = visitSched.wrkID
GROUP BY workerNumber, workerName

I get the proper columns grouped, but am getting 400,000 as the count when it
should be 1500-2000.
Any ideas how to make this return the results?