> 4A3*(
/0DTimes New Roman|dv0|(
0DArialNew Roman|dv0|(
0h" DSymbolew Roman|dv0|(
0h
`.
@n?" dd@ @@``x8' c$@{uʚ;2Nʚ;g4=d=dv0pppp@<4!d!d`
0,<4dddd`
0,<4BdBd
0,?%
?Issues related to quantitative methods and data analysis in PER@@(IDavid E. Meltzer
Iowa State University
Richard R. Hake
Indiana University2J IMethodological Issues that are Always Relevant(implicitly or explicitly):J
vValidity
Reliability
Statistical Significance
Pedagogical Significance
Representativeness of sample
Reproducibility
(vvHValidity Does your instrument provide data that could actually answer your research question?
Sample Flaw: To answer question, Do students understand concept A?, give them problem requiring knowledge of concepts A, B, C, and D.
Previous knowledge of specific mathematical tools or formal representations may need to be considered
WfWA6 fReliability~Is instrument internally consistent, that is, do different components of instrument measure (more or less) the same property?
~@-ReliabilityIs instrument internally consistent, that is, do different components of instrument measure (more or less) the same property?
If you made the same measurement again (all conditions apparently identical), would your instrument yield the same result?
@-Reliability~Is instrument internally consistent, that is, do different components of instrument measure (more or less) the same property?
If you made the same measurement again (all conditions apparently identical), would your instrument yield the same result?
Would minor variations in your test items (e.g., slight contextual or representational changes) lead to large variations in results?
$PZ-ZStatistical Significance$Is there a substantial probability ( 10%) that your result might have occurred purely by chance?
Do you have a measure of variance, or can one be estimated?
Can standard statistical tests be applied?
Parametric [e.g., linear correlation] and non-parametric [e.g., ranking]DcgI$?gI Pedagogical SignificanceIs the observed effect likely to be of practical significance in classroom?
Are there cost-benefit relationships implied in the magnitude of the observed effect?
Representativeness of Sample
`Is your sample representative of larger group from which it is (implicitly or explicitly) drawn?aaRepresentativeness of Sample
eIs your sample representative of larger group from which it is (implicitly or explicitly) drawn?
Are samples from different groups being compared equivalent in all respects except variable being investigated?
What are possible relevant variables?
How have you controlled these variables?
Have you done random selection? If not, what alternatives were used?* ReproducibilityDid you repeat experiment?
Did anybody else repeat the experiment?
Are your results substantially different from what others have observed, or are they otherwise very surprising?
If so, better check again!& ` ` ̙33` 333MMM` ff3333f` f` f` 3>?" dd@$|?" dd@ " @ ` n?" dd@ @@``PR @ ` `p>>$(
6 P
T Click to edit Master title style!
!
0p
RClick to edit Master text styles
Second level
Third level
Fourth level
Fifth level!
S
0L ``
X*
0 `
Z*
0 `
Z*H
0h ? ̙33 Default Design 4$(
4r
4 Sv0P0
r
4 Sdwp`
H
40h ? ̙33
0$(
r
SPP
r
S
H
0h ? ̙33
@0(
x
c$P
x
c$D
H
0h ? ̙33
P$(
r
SP
r
S§
H
0h ? ̙33
`0(
x
c$ǧP
x
c$xȧ
H
0h ? ̙33
p0(
x
c$\ЧP
x
c$ѧ
H
0h ? ̙33
$(
r
S$קP
r
Sק
H
0h ? ̙33
$$(
$r
$ SP
r
$ S
H
$0h ? ̙33
($(
(r
( SP
r
( Sx
H
(0h ? ̙33
,0(
,x
, c$P
x
, c$컧
H
,0h ? ̙33
0$(
0r
0 S0oP
o
r
0 Soo
H
00h ? ̙33r82"#%')+-/13$
x5
t(
On-screen ShowIowa State University^'
Times New RomanArialSymbolDefault Design@Issues related to quantitative methods and data analysis in PERJMethodological Issues that are Always Relevant (implicitly or explicitly) ValidityReliabilityStatistical SignificancePedagogical Significance Representativeness of SampleReproducibilityFonts UsedDesign Template
Slide Titles
Slide Titles (_^David E. MeltzerDavid E. Meltzer&.]H& --C~4-- dv0|(
0h
`.
@n?" dd@ @@``x8
c$@{uʚ;2Nʚ;g4=d=dv0pppp@<4!d!d`
0,<4dddd`
0,<4BdBd
0,?%A
?Issues related to quantitative methods and data analysis in PER@@(IDavid E. Meltzer
Iowa State University
Richard R. Hake
Indiana University2J IMethodological Issues that are Always Relevant(implicitly or explicitly):J
vValidity
Reliability
Statistical Significance
Pedagogical Significance
Representativeness of sample
Reproducibility
(vvHValidity Does your instrument provide data that could actually answer your research question?
Sample Flaw: To answer question, Do students understand concept A?, give them problem requiring knowledge of concepts A, B, C, and D.
Previous knowledge of specific mathematical tools or formal representations may need to be considered
WfWA6 fReliability~Is instrument internally consistent, that is, do different components of instrument measure (more or less) the same property?
If you made the same measurement again (all conditions apparently identical), would your instrument yield the same result?
Would minor variations in your test items (e.g., slight contextual or representational changes) lead to large variations in results?
$PZ-ZStatistical Significance$Is there a substantial probability ( 10%) that your result might have occurred purely by chance?
Do you have a measure of variance, or can one be estimated?
Can standard statistical tests be applied?
Parametric [e.g., linear correlation] and non-parametric [e.g., ranking]DcgI$?gI Pedagogical SignificanceIs the observed effect likely to be of practical significance in classroom?
Are there cost-benefit relationships implied in the magnitude of the observed effect?Representativeness of Sample
eIs your sample representative of larger group from which it is (implicitly or explicitly) drawn?
Are samples from different groups being compared equivalent in all respects except variable being investigated?
What are possible relevant variables?
How have you controlled these variables?
Have you done random selection? If not, what alternatives were used?* ReproducibilityDid you repeat experiment?
Did anybody else repeat the experiment?
Are your results substantially different from what others have observed, or are they otherwise very surprising?
If so, better check again!&rJ
J^
t(_5David E. MeltzerDavid E. Meltzer
6 !"#$%&5789:;<=>?@CDEFRoot EntrydO)=D$@B
Current User4PSummaryInformation( PowerPoint Document(^DocumentSummaryInformation8'
/0DTimes New Roman|dv0|(
0DArialNew Roman|dv0|(
0h" DSymbolew Roman|dv0|(
0h
`.
@n?" dd@ @@``x8!
c$@{uʚ;2Nʚ;g4=d=dv0pppp@<4!d!d`
0,<4dddd`
0,<4BdBd
0,?%@
?Issues related to quantitative methods and data analysis in PER@@(IDavid E. Meltzer
Iowa State University
Richard R. Hake
Indiana University2J IMethodological Issues that are Always Relevant(implicitly or explicitly):J
vValidity
Reliability
Statistical Significance
Pedagogical Significance
Representativeness of sample
Reproducibility
(vvHValidity Does your instrument provide data that could actually answer your research question?
Sample Flaw: To answer question, Do students understand concept A?, give them problem requiring knowledge of concepts A, B, C, and D.
Previous knowledge of specific mathematical tools or formal representations may need to be considered
WfWA6 fReliability~Is instrument internally consistent, that is, do different components of instrument measure (more or less) the same property?
If you made the same measurement again (all conditions apparently identical), would your instrument yield the same result?
Would minor variations in your test items (e.g., slight contextual or representational changes) lead to large variations in results?
$PZ-ZStatistical Significance$Is there a substantial probability ( 10%) that your result might have occurred purely by chance?
Do you have a measure of variance, or can one be estimated?
Can standard statistical tests be applied?
Parametric [e.g., linear correlation] and non-parametric [e.g., ranking]DcgI$?gI Pedagogical SignificanceIs the observed effect likely to be of practical significance in classroom?
Are there cost-benefit relationships implied in the magnitude of the observed effect?
Representativeness of Sample
`Is your sample representative of larger group from which it is (implicitly or explicitly) drawn?aaRepresentativeness of Sample
eIs your sample representative of larger group from which it is (implicitly or explicitly) drawn?
Are samples from different groups being compared equivalent in all respects except variable being investigated?
What are possible relevant variables?
How have you controlled these variables?
Have you done random selection? If not, what alternatives were used?* ReproducibilityDid you repeat experiment?
Did anybody else repeat the experiment?
Are your results substantially different from what others have observed, or are they otherwise very surprising?
If so, better check again!&r5
5J
t(
/0DTimes New Roman|dv0|(
0DArialNew Roman|dv0|(
0h" DSymbolew Roman|
!"#$%&()*+,-./01235Oh+'0\ `h
Issues that are always relevantDavid E. MeltzerlwaDavid E. Meltzerlwa21iMicrosoft PowerPoints r@*V@/p@=D$@2G2Ct 7&v &&#TNPPX2OMi
&
TNPP &&TNPPv
--- !v---&A]&/gw@J
IwIw0- &.]H& --C~4-- @"Arialw@
IwIw0- .62
Issues related to quantitative
. .72
A methods and data analysis in PER)
!!#.--M-- @"Arialw@J
IwIw0- .2
w*David E. Meltzer !
.@"Arialw@
IwIw0- .'2
/Iowa State University
.@"Arialw@J
IwIw0- .2
*Richard R. Haker
.@"Arialw@
IwIw0- ."2
DIndiana University
.--"System
0-&TNPP &՜.+,0