Developing JobRelated Test Materials Using the CATS SYSTEM

Developing Job-Related Test Materials Using the CATS SYSTEM CODESP Automated Test System

WHAT IS CODESP? Cooperative Organization for the Development of Employee Selection Procedures • A nonprofit consortium of over 700* public school districts, community colleges and charter schools which was formed in 1973 to combine resources to develop classified test materials • Products include: Written Test Items, Interview Questions, Supplemental Application Forms, Performance Exams, Sample Tests (including Instructional Aide) and Writing Exercises * Includes over 300 small districts who have free membership through their County Office of Education because their ADA is less than 1, 000.

How Can CODESP Help? • CODESP’S Item Bank, soon to be called (CATS), contains over 17, 000 items covering most job classes found in public school districts • A job-related test can be easily developed from the items CODESP provides • Multiple types of assessment tests are available

How Can You Help? You can be instrumental in creating job-related exams By Conducting SME item reviews and Job Analysis And Notifying CODESP regarding SME Review and Job Analysis results to update test items

If Insufficient Attention is Given to the District’s Employee Selection Process: • Employment-related lawsuits may occur: a single legal challenge by an aggrieved applicant can cost thousands of dollars in government fines and tort liability • Wasted recruitment time & money when positions must be reopened • Wasted salary and benefits to employees who are poor matches or who only stay a limited time • Wasted training dollars and time • Poor service to customers • Reduced morale among current employees who must work with non-qualified co-workers

Ideal Selection Results In Job-Related/Valid Selection Devices/Tests Accurately Measured Candidate Ability Increased Quality of Candidates Selected Successful Organizations

Content Validity • Referred to in the Uniform Guidelines and many court cases: – The content of the test must represent important job-related behaviors – Test items should be relevant to and measure directly important requirements and qualifications for the job (KSAs) – Also known as “job-relatedness” or face validity • Achieved through job analysis

Job-Relatedness • To achieve content validity each test item must be related to the job • If not, the validity of the selection procedure is jeopardized • Validity legally defensible

Court Case Griggs v. Duke Power Outlined the need for conducting an analysis of the job for which a selection procedure has been developed • Artificial barriers which are not job-related should be removed from the selection process • Tests should assess minimum requirements needed at the time of hire • Tests should measure how the person will potentially perform on the job, not the person in the abstract

The Problem… • It is difficult for HR professionals to be “experts” and make judgments about the job relatedness of test items in all job subject areas

The Solution… • Gather information about the job and test items from “SMEs” Subject Matter Experts

How Do We Find SMEs? • Use existing knowledge of district personnel • Contact schools/department heads for suggestions • Build lasting, positive partnerships with departments and SMEs for future projects • If you are unable to find SMEs, contact CODESP, or other member districts directly

Steps To Take When Working With SMEs To Develop A Selection Test: 1) Conduct a job analysis or check to see if major changes have occurred since the last job analysis (w/SME) 2) Review current Job Description 3) Obtain potential test items (create them yourself or obtain them from CODESP) 4) Review potential test items (w/SME) 5) Select the best items for the test 6) Create and Print test online using CATS

Purpose Of Job Analysis • Ensures that your current job description is correct and up to date • Identify the knowledge, skills, and abilities (KSAs) required at the time of hire to perform the job successfully • Is required to increase the validity of the test items

Job Analysis Process • CODESP’s new Job Analysis Tools: – Job Analysis Questionnaire – Job Description Builder – Products will be available Summer ’ 07 – See Job Analysis Basics Training under Resources / Training Presentations – See Resources / Job Description Builder and Questionnaire

Item and Job-Level Match • It is important to determine whether the knowledge measured by an item is essential for the successful job performance of: • entry level employees • intermediate level employees • high level employees • Items should only test knowledge that is essential at the time of hire

Item Writing Guide The following are a series of hints and suggestions to help your Subject Matter Experts construct or review test questions: • Write out the full title when describing objects and procedures. Only use acronyms when you are positive that your audience knows exactly what you are describing. When in doubt, spell it out. • Clearly indicate the correct answer for each question. There should only be one “best” answer.

• Stay away from slang because these words can be open to different interpretations • Try to avoid distracters that include words like “always” and “never” because they are generally false statements as well as words like “some” and “usually” because they are generally true statements • Ask questions that have definite answers so that most experts will agree on the correct response • Try to have a balance of easy, moderate, and hard questions. All disciplines need to be balanced with respect to difficultly level. • Don’t ask for details that only individuals who have worked in that particular department would know. • Watch for hints in the stems (e. g. , synonyms). • Do not create questions that have negative stems (“Which one of the following is not…”), unless you are addressing a certain action that all employees need to should be never done. • Questions that have complex answers are less efficient and provide less reliable results.

PROBLEM ITEMS • • • Not applicable to job/field Terminology too complex Outdated Incorrectly keyed Question Unclear Choice Unclear Other choices are also correct Other choices are too obvious Choices are not parallel

SME Test Item Security • Use A SME Background/Security information form (found under Resources / Exam Planning) and keep it in the test file • Make sure that CODESP test materials are stored in a secure location

CODESP CATS SYSTEM CODESP Automated Test System

Getting Started To get started, launch an Internet Explorer session and enter the following web address: http: //www. codesp. com. Then, press the Enter key on your keyboard. This section will cover the following topics: Website Navigation Registering for Services Logging In Logging Out

Navigating the CODESP Website The CODESP banner and the navigation menu make information readily accessible, making website navigation a breeze. • Some information is only accessible to registered members who have logged in.

Helpful Hints • Because this website makes use of pop-ups, it is important to disable any pop-up blocker software that may be installed on your machine. You may be able to disable popups on our website without disabling pop-up blocker software entirely depending on the software installed. • Graphical icons are often used to simplify website interface. Simply, hover your mouse over the icon to view the functionality.

you Member Name Navigating the Website Your member start page provides you with a customized view of CODESP training events, your class registrations, newsletters, announcements and your test material request status.

Currently Registered Members • The next several slides provide instructions on registering for CODESP • If you are already registered, skip to Slide 55

Registering for Services Becoming a CODESP member is easy and offers many benefits. With CATs CODESP members have access to: • Test Materials • Tutorials • Job Analysis Questionnaire • Job Description Builder • Training • HR Resources Learn more about the different memberships CODESP offers by clicking on the Membership tab found on the toolbar.

Registering for Services Continued To become a member of CODESP, click on Register Now and complete the following requirements: • Application for Services • Security Agreement • Membership Agreement * • Purchase order or check for the full amount of the membership fee * *Not needed by districts and Charter Schools with an ADA of 1, 000 or less if County Office is a member

New Members Register on the CODESP Home Page Once you are registered you will receive a Member Log -in Name and Password so that you can access the secure areas of the website.

To begin the registration process, select the desired membership. Mid-Year is only available from Jan – June and requires a commitment for a full-year membership the following year. * Districts and Charter Schools with an ADA less than 1, 000 do not need to complete these requirements, as long as their County Office is a CODESP member. They only need to complete an application and security agreement. Their membership is free.

The CODESP Registration Wizard guides you through each step of the registration process: Step 1: Application for Services • Select your County and District from drop down menu. • Verify the address. • If address is incorrect, make the appropriate changes. • Complete the rest of the form, including a primary and secondary contact person(s). • Click Next to move to the following screen.

• If your district does not appear on the drop down menu, click What if my district is not listed above? • Fill out the form as seen in the next figure and click the submit button. You will be contacted by a CODESP representative in order to verify your request. Once CODESP has added your district, you will be able to complete the registration process.

Step 2: Membership Agreement • Read the Membership Agreement thoroughly. • To comply, click “I agree with the terms and conditions as stated above. ” • Click Next to move to the following screen. Note: You will not be able to continue registering unless you accept the terms.

Step 3: Security Agreement • Similarly to Step 2, read the Security Agreement thoroughly. • To comply, click “I agree with the terms and conditions as stated above. ” • Click Next to move to the following screen. Note: You will not be able to continue registering unless you accept the terms.

Step 4: Registration Information Review • Verify that the registration information provided is correct. • Click Edit to revise the information. • Click Next to move to complete the CODESP Registration Wizard.

Step 5: Print, Sign and Fax Agreement CODESP is legally obligated to keep a signed contract on file for every CODESP member. • • • Please print all forms. Sign them. Fax or mail them to the CODESP office: 20422 Beach Blvd. Suite 310 Huntington Beach, CA 92648 Fax: (714) 374 -8225

The Registration Wizard ends at Step 5, but remember, in addition to sending the signed agreements, don’t forget to send your membership payment. Checks and purchase orders are accepted. Forward an Invoice to your Business Office. An invoice is available in the Document Library. Once CODESP has received the membership payment in full and the signed agreements, CODESP will notify the primary contact and your district will be given a District Administrator account. The District Administrator will be assigned a login name and password. To learn more about the District Administrator role and responsibilities, see section titled, District Administrator Account.

Logging In If your account has already been established, click Log In. Otherwise, contact your District Administrator in order to set up an account.

Logging In Continued First time users must agree to the terms of the security agreement even if one has been completed by other district users:

Logging Out Throughout the course of the day, there will be times in which you will not need access to the system— lunch time, breaks, meetings, etc. In these instances, you will need to exit the system. In order to maintain the highest level of security, end your session by clicking on Log Out. To Keep the Site Secure – Always Log-Out When You are Not at Your Workstation

My Profile Your profile can be viewed by clicking My Profile, located on the navigation bar. You will notice certain fields are editable, while others are for viewing purposes only. For security reasons, a person cannot change their own Status or Security Role. This data can only be changed by another District Administrator. Never use another person’s log-in name and password

Membership District Directory Your district will benefit from networking with other districts. The District Directory serves as a CODESP phone book and lists all the districts that are CODESP members. To access the District Directory, click District Directory located on the navigation bar. The searching tools provided make it easy to find a district’s contact information. You can search by text or by county. Searching by Text • Click Text as it is highlighted in the following figure. • Enter keyword(s) in the Search For field. • Click drop down menu to view the searchable fields. • Highlight the field you wish to perform a search on. • Click Search Now

Search by County 1. Click County as it is highlighted in the figure below. 2. Click drop down menu to view list of counties. 3. Highlight the county you wish to find. 4. Click Search Now. By default, your search results will be displayed 5 districts per page, but conveniently, you can change the view to display 10 districts per page, or if you prefer, you can select Show All to view all the CODESP members.

District Directory Continued The results page displays information about the district, including: • District Name • Address • Web Address (if information was provided) The icons on the right-hand side serve a purpose as well. To find out their functionality, simply hover your mouse over each icon and a description will appear. By clicking these icons, you can: • View Contact Information • Email Primary Contact

CATS The term CATS is used on the CODESP website frequently and is also found throughout this guide. CATS is an acronym that stands for CODESP Automated Test System. This section will cover the following topics: • Requesting Test Materials • Picking up Test Material Packets • Building your Test • Reusing Tests • Reports

Requesting Test Materials The CATS Request Form is an order form used to order test materials such as multiple choice test questions, supplemental application forms, and interview questions, etc. . To make a request, simply follow each step: • Click on CATS Request Form listed on the navigation bar. • Read the CATS Information Request Directions. • Click on Fill Out Request Form.

CATS Request Form Cont’d • The CATS Information Request Form screen will appear. An example is listed below. • Enter a job title. • Select a month and day from the drop down menu to indicate when the materials are needed by. • We require a MINIMUM of 7 Business Days • Provide CODESP with specific information and instructions about the position, so CODESP can collect test materials that are suitable for your district. • Click Submit when you are ready to submit your request.

CATS Request Form Cut and paste your job description into the Comments Box. Provide details if your description is vague. For example: Name the specific software required if your job description only states: Computer skills required.

Picking up Test Material Packets • Based upon the job description and/or the special instructions listed on the CATS Request Form, CODESP will create a packet of test questions from their question database and send them to your account. The packet of testing materials will be accessible online. • When the packet of test materials has been completed, CODESP will send an email notification to the individual specified as the contact person on the CATS Request Form. The email subject line will read: CATS Materials Request Confirmation.

Email Notice You Will Receive • Your CATS Request with the job title of 'Another Sample Request Worker' has been processed and your materials packet is ready for your review. (ignore numbers identifiers for packets) To review your materials, log in to Codesp and click on the 'Create a Test' link from the left-hand navigation menu

Picking up Test Material Packets Cont’d To pick up your packet: • Log onto your account. • Click the hyperlink under My Request Status and the following will appear.

Tracking Your Request You can also track the status of your request by looking at the My Request Status section found on the start page. Your request will be displayed as one of the following: • New: Request has not been processed by CODESP. • In progress: Request is currently being processed by CODESP, but it is not complete. • Finished: Request has been completed. Request now called packet. Click on hyperlink. It will be stored under: Create a Test on the webpage.

Training Registration Reminders are listed on your CATS home page. Also, My Request Status lets you know when your Request is Ready. Note: The My Request Status section will only display the status of your requests that have been made in the last 30 days.

To view all packets that have been processed, click on Create a Test found on the navigation bar.

Building Your Test Now it is time to generate your test! To begin the CATS Test Generator, follow these steps: • Click on Create a Test, which can be found on the navigation menu. • Choose the packet you would like to work on by clicking the position title.

Building your Test Cont’d • Provide a title for the test. • Allocate how much time the candidate will be given to take the test. • Click Next when you are ready to move onto the next screen.

Custodian

Building your Test Cont’d Review all questions in packet. Note: The correct answers are indicated with a green check mark. • To add question to your test, check the box under Add to Test. • Optional: Sort questions in the order you wish them to appear on test (change test number). • Click Create a Test to complete this process. • If you are not satisfied with the materials in the packet, call us first and we will advise you to click Send Back or to submit a new Request.

All Questions which begin with an F require you to click on the Corresponding Figure above the item (Figure lose their item numbers during the print process) See CATS Tutorial for more details.

Building your Test Cont’d • If you Send Back the items explain to CODESP why these materials did not meet your needs and what subject area you need additional questions from. • CODESP will then rebuild the packet based on your comments. If it is a Totally Wrong Packet-Call us we will advise you if you should submit a new Request. • Send Backs can be avoided if you provide us with accurate information about the job on the Request Form.

Reusing Tests After you have generated a test, the test will be stored in the Test Library. Each test can be reused by clicking on the printer icon. The Red X will delete the test. Only delete tests that were created as practice. Tests that were administered should remain in your Library, a long with a hard copy in a LOCKED cabinet. We suggest regenerating your test each time you administer it to receive the latest updates.

Printing a Test • By clicking on the Printer Icon you can print the test from your desk top. • Follow the instructions as you go through the print process. • Enter the time you want for your test. • Make sure you Click All Pages. • Make sure your title is correct. • Check the test before administering – Are all pages printed – Are the figures easy to read – Did all of the figures appear above the corresponding items.

Interviews, Supplementals, Performance Tests and Writing Exercises Samples are available in Word format in the Resource Library. • Scroll down to the type of test. • Click on the subject area. • Click on the title which most closely matches the title of the position you are requesting sample test materials for. • Open a separate Word document and copy and paste the questions from the samples provided to your blank Word document.

Resource Library Click on the type of documents or test materials needed to expand the selection.

CODESP SAMPLE INTERVIEW QUESTIONS: ACOUNTING TECHNICIAN/AUDITOR SAMPLE INTERVIEW QUESTIONS 1. Describe your experience auditing business records to determine unemployment insurance premiums, liabilities, and district's compliance with federal and state laws. How extensive were your audit reports? 2. Describe your experience reconciling District's internal records with the County PBAS System. (Add whatever system is required for this position). 3. Describe how you have utilized your knowledge of the State of California Accounting Manual for Local Education Agencies in conducting school district audits. 4. Describe your experience auditing and reconciling state and county revenue limit reports.

After copying and pasting into your own Word Document you may edit the questions to fit your districts needs such as the following sample using the first 2 questions from the previous slide: ACCOUNTING TECHNICIAN 1. Describe your experience auditing business records to determine unemployment insurance premiums, liabilities, and district's compliance with federal and state laws. How extensive were your audit reports? 2. Describe your experience reconciling district's internal records with the County SAS System.

Pre-Assembled Tests Pre-Assembled tests for positions such as Custodian (entry and experienced), Food Service Worker and Food Service Cashier, Clerk, Instructional Aide, and Secretary are be available for viewing in the Resource Library. You can use them as is or order a custom test by completing a CATS Request Form.

District Administrator Account District Administrator accounts are unique. A person who has District Administrator access has additional features on their navigation menu, which gives them access to: Add/Edit Profiles Edit District Profile In the illustration that follows, it is pointed out that all the links below the dashed line are features only accessible by people who have a District Administrator account. Links above the dashed line are available to all users.

This section will cover the following topics: Add New User Profile Edit District Profile

Add New User Profile Upon registering for services, CODESP creates one District Administrator account. Once this account has been created, it is the District Administrator’s responsibility to create additional accounts for other district employees. Add User Profile 1. Click on Add/Edit Profiles 2. Click on the Add New User button located in the bottom right-hand corner. 3. The Add New User screen will appear.

Add User Profile Cont’d 1. On the Add New User screen, assign user a login name and password. 2. Enter user’s contact information. 3. Define user’s Security Role by selecting Application User or District Administrator (does not have to be an actual “Administrator”. 4. Click Update Profile to save.

Edit User Profile The District Administrator is charged with maintaining user profiles. As a reminder, listed below are common occurrences that should raise a red-flag to go in and amend the user’s profile: Person changes positions • Person retires Contact information changes Person changes their name Person will be taking a test for another job in district Edit Profile 1. Click on Add/Edit Profile located on the navigation bar.

Edit Profile Continued 1. Make appropriate changes. 2. Click Update Profile.

Edit District Profile The District Administrator is also responsible for keeping their district contact information current. This information is found by clicking Edit District Profile. After you have made your changes, click Update Profile.

CATS Training • CATS Tutorials with more detailed instruction are located under Resources / Tutorials • Give us a call if you can’t find the answer on-line: 714 -374 -8644 We are continuously researching and/or adding new features

QUESTIONS? ? Email: tests@codesp. com www. codesp. com